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Abstract 

We present a new approach to distributing 
processed telemetry data among spacecraft 
flight controllers within the Control Centers 
at NASA’s Johnson Space Center. This ap- 
proach facilitates the development of appli- 
cation programs which integrate spacecraft- 
telemetered data and ground-based synthe- 
sized data,, then distribute this information 
to flight controllers for analysis and decision- 
making. 

The new approach combines various dis- 
tributed computing models into one hybrid 
distributed computing model. This model 
employs both client-server and peer-to-peer 
distributed computing models cooperating to 
provide users with information throughout a 
diverse operations environment. Specifically, 
it provides an attractive foundation upon 
which we are building critical real-time mon- 
itoring and control applications, while simul- 
taneously lending itself to peripheral applica- 
tions in playback operations, mission prepa- 
rations, flight controller training, and pro- 
gram development and verification. 
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We have realized the hybrid distributed 
computing model through an information 
sharing protocol. We shall describe the mo- 
tivations that inspired us to create this pro- 
tocol, along with a brief conceptual descrip- 
tion of the distributed computing models it 
employs. We describe the protocol design in 
more detail, discussing many of the program 
design considerations and techniques we’ve 
adopted. Finally, we describe how this model 
is especially suitable for supporting the im- 
plementation of distributed expert system ap- 
plications. 

1 Introduction 

The past approach to spacecraft data distri- 
bution in the Mission Control Center (MCC) 
used a centralized computing model. Main- 
frame computers processed the incoming 
telemetry and trajectory data, applied var- 
ious computations to this data, then dis- 
tributed this information to hundreds of flight 
controller displays. This approach worked 
well for many years, but its cost, inflexibil- 
ity and resistance to change have become 
unappealing. Meanwhile, advancing work- 
station and networked, distributed comput- 
ing technologies which overcome these limi- 
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tations grew more compelling. Accompany- 
ing these advancements were new software 
development packages, man-machine inter- 
faces, and artificial intelligence technologies. 
The application potential of these exciting 
technologies inspired flight controllers to as- 
sume more interactive roles in the develop- 
ment of computational tools which enhance 
their spacecraft operations capabilities. The 
desire to implement these enhancements en- 
couraged a revolution in the MCC computing 
architecture. 

The MCC began the transition from cen- 
tralized to distributed computing several 
years ago. The current MCC employs both 
configurations simultaneously, running work- 
station and mainframe applications side-by- 
side. As the MCC evolves into the Control 
Center Complex (CCC) for dual Space Shut- 
tle and Space Station operations, the work- 
stations and network will become the princi- 
pal computing platform. 

The current dual-configuration mode of op- 
erations in the MCC has afforded flight con- 
trollers an opportunity to prototype new op- 
erations applications. Until recently these ef- 
forts have focused on the development of real- 
time monitoring and fault detection and diag- 
nosis applications for individual flight control 
disciplines. These efforts have been success- 
ful in demonstrating the enhanced capabil- 
ities provided by the distributed computing 
platform. Moreover, they have encouraged 
the pursuit of distributed applications which 
span several flight control disciplines. These 
distributed applications exploit networking 
capabilities to provide inter-user information 
sharing. 

The new distributed computing model we 
describe herein was created to support and 
encourage this inter-user information shar- 
ing. The information sharing protocol (ISP) 
was designed to provide elegant computing 
techniques which enable workstation applica- 
tions to communicate with each other. This 


protocol also supports inter-host communi- 
cations on a heterogeneous platform comply- 
ing with industry-standard operating systems 
and networking protocols. Consequently, the 
ISP provides users with a consistent commu- 
nications interface for all of the various oper- 
ations, training, and development platforms 
they use in everyday business; lack of such 
an interface has caused many problems in the 
past. 

There were several motivations that led 
to the development of the ISP. First, there 
has been a strong desire to reduce the man- 
power required for real-time Space Shuttle 
operations. This reduction in manpower ne- 
cessitates the development of enhanced com- 
puter programs which can perform many of 
the real-time data monitoring, fault detection 
and diagnosis, and planning tasks previously 
performed by flight controllers. These de- 
velopments have also spawned many projects 
pursuing “intelligent systems” which meet or 
exceed the flight controller’s reasoning ca- 
pabilities. Many such systems have already 
been deployed in the MCC, but there are 
many more of these intelligent systems ap- 
pearing on the horizon. The potential for 
these systems to capture precious spacecraft 
operations expertise is a clear motivating fac- 
tor in our work. Second, the increasing num- 
ber of workstation programs being developed 
to enhance operations has been accompa- 
nied by a significant growth in software de- 
velopment and maintenance costs. The de- 
velopment of the ISP was meant to restrain 
this growth by providing a package of dis- 
tributed computing tools which support the 
basic needs of all flight control disciplines. 
These tools reduce both development and 
testing costs. Third, the use of workstations 
outside of the MCC has increased tremen- 
dously over the last few years. Flight con- 
trollers now find themselves performing many 
of their mission preparation, training, and 
software development tasks in facilities lo- 
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cated in their offices or laboratories around 
the center. Since many of the computer pro- 
grams they use must work in all of these facili- 
ties, there is a strong movement toward stan- 
dardization. This movement is encouraging 
the use of industry-standard hardware and 
commercially- developed software to reduce 
development and maintenance costs. The 
ISP promotes these standards while provid- 
ing a layer of customizable telemetry data 
distribution tools for program development 
in a heterogeneous facilities environment. Fi- 
nally, drawing from each of these individual 
motivations, we envision a tremendous util- 
ity in the deployment of distributed expert 
systems. The ISP was designed to provide a 
mechanism for these expert systems to com- 
municate with each other. 

2 Distributed 
Computing 

A distributed computing architecture is a 
programming model in which computing re- 
sources can be allocated in various ways. 
Typically this involves a network of high- 
performance workstations providing informa- 
tion to a user. There are two predominant 
models of how to distribute the available com- 
puting resources. A client-server computing 
model refers to a special case of distributed 
computing in which an application is divided 
into separate client and server processes. The 
client process makes requests of the sever pro- 
cess. The server process receives requests 
from one or more clients and performs some 
action on their behalf. Servers can provide 
resource sharing by servicing many clients. 
Frequently, the client and server pieces are 
running on different machines; however, this 
need not be the case. In a peer-to-peer com- 
puting model, the distributed components of 
the application act as independent equals, 
communicating with one another to accom- 


plish a common goal. 

The distributed computing model realized 
in the ISP reaps the benefits of both of these 
models. At a global level, the goal of the 
application is to distribute telemetry data to 
flight controllers. In the client-server role, the 
primary purpose of the server part of the ap- 
plication is to process telemetry data from a 
certain data source. The clients request this 
processed data from the servers. In the peer- 
to-peer role, the servers allocate computing 
resources and communicate with one another 
to provide their clients with data collected 
from several data sources. The combination 
of these models provides a system of cooper- 
ating agents which supports a wide variety of 
potential applications. 

The user assigns each server to process 
a portion of data available from the data 
source. The server provides this data to its 
clients on a change-only basis; i.e. a client 
will receive data only when it changes from 
its previous value. Since there are a variety of 
data sources available, we have written a col- 
lection of server programs each of which has 
been uniquely outfitted to process a particu- 
lar data source. The data sources include two 
real-time data streams and various sorts of 
recorded telemetry files. The server prepro- 
cessing includes reading data from the data 
source, performing noise filtering and other 
preprocessing on the data, determining what 
data has changed since the last cycle, and 
then distributing the changes to the clients. 
Each server is responsible for any time syn- 
chronization and polling tasks. 

The user may run clients that receive data 
from the server and generate new results to 
supplement the telemetry data. Therefore, 
the servers also perform the task of reading 
and distributing this client-synthesized data. 
Clients that synthesize data for other clients 
are said to be publisher clients, while clients 
that request any sort of data are said to be 
subscriber clients. Since each server “owns” a 
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portion of the data source, the servers can in- 
teract in the peer-to-peer role to acquire data 
for clients which subscribe to data owned by 
other servers. In this situation, one server be- 
comes a subscriber of another server. Along 
with resource allocation and data distribu- 
tion, this technique provides a way to enforce 
data security. 

Although there are unique servers for each 
data source, the data source is transparent 
to a client. Using the ISP, a client receives 
data the same way regardless of where the 
data originated. This feature provides a 
high degree of portability between comput- 
ing platforms, and insulates the clients from 
the nuances of the data source. Essentially, 
this means tha f a client program can be de- 
ployed against a variety of data sources with- 
out change. This data-source independence 
simplifies application program development 
efforts and reduces the costs associated with 
software testing and certification. 

3 Design Considerations 

This section describes a few of the considera- 
tions we have made in the design of the ISP. 
Most of these considerations reflect the moti- 
vations established above, while others reflect 
the specific nature of the telemetry data pro- 
cessing problem. We discuss here only the 
considerations which we believe to be Space 
Shuttle-independent. Primarily, we discuss 
the construction of our hybrid distributed 
computing model by presenting the attrac- 
tive features of the two underlying models. 

3.1 Client-Server Model 

The client-server model is the most influential 
part of the ISP model. Using this model, we 
separate the telemetry data source from the 
client programs. We create server programs 
to manage the data source and distribute pre- 


processed data, and we create client programs 
to do something useful with this informa- 
tion. Usually the client programs perform 
analysis of the information and display the 
results to the flight controllers. This separa- 
tion of tasks is crucial to the hybrid model: 
it provides the cability to support a vari- 
ety of data sources and various computing 
platforms with the minimum number of pro- 
grams; and it allows user to run various com- 
binations of clients in servers in different con- 
texts, such as playbacks or offline simulations 
and training cases. 

The ISP servers provide information to 
their clients on a change-only basis. This 
means that the client programs may operate 
in an event-driven fashion, instead of having 
to poll the data stream continuously . 1 When 
new data is made available to the client, that 
data is considered an interrupt “event' that 
the program must handle. The client han- 
dles this event by reading the data, process- 
ing and perhaps displaying it, then return- 
ing to its wait state to receive subsequent 
events. This provides an efficient allocation 
of computing resources and reduces the effort 
necessary to build real-time computer pro- 
grams. Furthermore, to make better use of 
the distributed computing platform, we can 
divide this effort among several functionally- 
equivalent servers processing a different sub- 
set of the full telemetry stream. By not ex- 
traneously reprocessing the unchanged values 
every second, each client realizes a significant 
computational resource savings and, perhaps 
more importantly, appears more responsive 
to user interaction. 

We assume that there will normally be 
many servers running. Each server will pro- 
cess a portion of the full data stream. At ini- 
tialization time, the server assumes an owner- 
ship which identifies those symbols from the 

’Historical Space Shuttle mission data shows that 
during on-orbit operations only 10-15% of the 25, 000 
telemetry values change during any second. 
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symbol dictionary that it becomes responsi- 
ble for. The server will acquire and process 
all of the symbols from the symbol dictionary 
which have the same ownership as the server. 
For example, the Propulsion (PROP) console 
server started with ownership prop will ac- 
quire from the data source all of the symbols 
in its dictionary with ownership prop. All 
PROP client programs need connect only to 
a prop server; similarly all Guidance, Naviga- 
tion and Control (GNC) console clients only 
need connect to a gnc server, and so on. 

Although the server’s primary purpose is 
to process telemetry data, the server is also 
capable of processing and redistributing data 
generated by publisher clients. This feature 
is vital to distributed client applications, par- 
ticularly fault detection and diagnosis appli- 
cations, which intend to share their results 
with other clients. Under the ISP, this dis- 
tribution occurs when the publisher client 
writes its results back to the server, which 
enqueues the information for any subscriber 
clients. Since the distribution mechanism is 
the same for any kind of data, the ultimate 
subscriber client will not be able to discern 
where the data originated, i.e, from teleme- 
try or from a publisher client. 2 

Finally, since the servers are meant to be 
background tasks, they do not have user in- 
terfaces. To provide server status informa- 
tion and control features, we simply employ 
a collection of client programs which act only 
as user interfaces to their server. These user 
interface clients don’t subscribe for any data 
from the server; instead, they read the server 
log files for status information, and they send 
the server control commands through packet 
messages. Furthermore, this design embodies 
an additional feature that the separated user 
interface can converse with any of the servers 

2 In fact, we have built a server which does not 

process a data source at all: it simply redistributes 
information generated by its publisher clients. We 
refer to this as a null data source server. 


without modification. 

3.2 Peer-to-Peer Model 

Commonly, clients from one flight control dis- 
cipline may need to process symbols “owned” 
by another discipline. We perform this ex- 
change with a server to-server, or peer-to- 
peer , interconnection. Fundamentally, this 
means that one server defers requests for sym- 
bols that it doesn’t own to another server. 
For example, if a PROP client program needs 
access to some GNC telemetry symbols, the 
prop server will defer requests for these sym- 
bols to the gnc server. In this situation, the 
prop server effectively becomes a client of the 
gnc server. The gnc server will send the re- 
quested data to the prop server, which in 
turn will forward the data to the prop client 
program. 3 This situation is depicted in fig- 
ure 1. The ownership field in the symbol 
dictionary provides some of the information 
needed to establish this connection. Basi- 
cally, it provides the name of the server which 
publishes that symbol. Therefore, when a 
client subscribes with its server for a symbol 
published by a different server, that client’s 
server looks for the publishing server running 
somewhere on the network. The ISP network 
registration service (NRS) is employed to per- 
form this search. Once connected, the defer- 
ring server forwards the symbol subscription 
to the peer server, while the peer server regis- 
ters the deferring server as its client. Symbol 
values will flow from the peer server to the 
deferring server then on to the client. We 
use this process to perform data security: by 
requiring configuration management of the 
symbol dictionaries, we can prevent clients 
from subscribing to symbols that they aren’t 
authorized to receive. 

3 Note that although they are independently pro- 
cessing different information, these two server pro- 
cesses are instances of the same program that were 
given different ownerships at run time. 
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Figure 1: Inter-operator data distribution model. The PROP client can receive both PROP 
and GNC telemetry data, but it must acquire the GNC-owned data indirectly from the GNC 
server. 


Telemetry LAN 





Figure 2: A distributed-resource situation. The DPS flight control discipline allocates a 
special server to the task of processing fault detection messages generated by the spacecraft 
software. This server distributes these messages to DPS clients directly, and to the PROP 
clients through a peer-to-peer connection. 
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These peer-to-peer connections also are 
important because they manifest how the 
model achieves global data processing. All 
of the processing necessary to make a par- 
ticular data value “useful” can be performed 
by the server or client that “owns” that 
symbol. All other clients which may need 
the data, but don’t own it, do not need 
to duplicate the processing; they simply re- 
quest it from the owner. This feature es- 
sentially replaces the centralized computing 
feature which performs all of the data pre- 
processing before passing the data on to user 
applications. 4 This enables a wide variety 
of distributed resource allocation schemes in 
which each server provides a different data 
processing function. Figure 2 suggests one 
such distributed-resource situation. 

3.3 Implementation 

In realizing this protocol we have employed 
many industry standards. Primarily, we 
allow Unix do the things it was meant 
to do, such as managing resources. We 
use the TCP/IP protocols for network ser- 
vices, and we use point-to-point connection- 
oriented stream sockets for interprocess com- 
munications. Since the majority of our facili- 
ties employ the X-Windows system, we mimic 
certain facets of the X toolkit. Each ISP 
event has an associated enumerated type def- 
inition uniquely identifying that event. Many 
events also have associated data structures 
containing event-relevant information, such 
as a telemetered sensor value and time stamp. 
All of the events are distributed via message 
packets. Each message includes a header and 
a body, and the body can be of arbitrary 
length. Like the X toolkit, the ISP toolkit 
provides a callback mechanism whereby the 

4 This is particularly important for data that is 
limit-sensed in some manner. The owner of the data 
applies the proper limits before forwarding the results 
to subscriber clients. 


programmer can register a series of functions 
to be called automatically upon arrival of 
a given event packet. Each registered in- 
stance of a callback function can have unique 
callback data to be passed to that function 
through the argument list. Essentially, this 
implementation allows programs built around 
the X protocol to use the same processing 
logic for “data” events that they use for “dis- 
play” events. 

When a client wishes to establish a session 
with a server, a variety of information must 
first be exchanged (figure 3 depicts the event 
exchanges). The client opens a socket connec- 
tion with a server, then issues a connection- 
request event to it. If the server receives and 
accepts the connection request, it will reply 
with a connection-accept event. Upon receiv- 
ing the connection-accept event, the client 
submits a series of subscription requests for 
all of the symbols it wants to process. The 
server validates all of these subscription re- 
quests and updates its internal symbol tables. 
When the client is done submitting subscrip- 
tion requests, it sends an event to enable the 
data stream. The server will then begin is- 
suing data-change events to the client as its 
symbol values change. The client will receive 
these events and process them, possibly send- 
ing some synthesized data back to the server 
for publication. When the session has com- 
pleted, the two programs exchange disconnec- 
tion events to gracefully terminate communi- 
cations and release resources. 

4 Distributed Expert 
Systems 

Some of the most exciting applications of 
the ISP technology lie in the support of dis- 
tributed expert systems applications. Tra- 
ditionally, these distributed expert systems 
have required that each component employ 
the same inference engine or expert system 
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Figure 3: Client-server event exchanges. 


shell. Using the ISP, however, we overcome 
this limitation by establishing a common in- 
terface enabling these expert system compo- 
nents to communicate with each other. Fur- 
thermore, since the ISP message packets can 
contain arbitrary data, a variety of informa- 
tion can be communicated. The information 
can be a sensor reading, a CLIPS fact, the 
results of a simulation or analysis, a fault 
message, and so on. These features promote 
a gradual development of heterogeneous dis- 
tributed expert systems using the most ap- 
propriate inference engine or shell for the 
task. 

Using the ISP, we have already begun to 
deploy some exciting distributed expert sys- 
tem applications for the CCC. We summarize 
a few of the highlights below: 

PRS We have already deployed the ISP 
within prototype applications of the Pro- 
cedural Reasoning System (PRS) [1, 2]. 
PRS is a real-time, multi-agent reasoning 
system capable of monitoring and con- 
trolling complex physical systems. The 
PRS development work performed for 
NASA has focused on the problem of 
handling spacecraft malfunctions and ex- 
ecuting diagnostic procedures, assisting 
the operator in monitoring execution of 


these procedures, and adapting them to 
new situations. The PRS agents con- 
verse with each other in a peer-to-peer 
fashion, distributing the workload and 
knowledge bases according to the var- 
ious tasks being performed by the ap- 
plication. These agents exchange infor- 
mation on a change-only basis, updating 
each other’s data bases and contexts in 
order to monitor and control the behav- 
iors of the physical system. Using the 
ISP, we empower sophisticated fault de- 
tection and analysis programs to provide 
status information to the PRS malfunc- 
tion management systems, and provide 
an interface through which we merge 
PRS results into conventional telemetry 
displays. 

VISTA Working with the Vista project 
team at the Rockwell Science Center 
Palo Alto Laboratory, we are develop- 
ing an integrated decision-theoretic ap- 
proach to the problem of managing the 
complexity of displays used in high- 
stakes, time-critical decision contexts [3, 
4]. As a side effect of this effort, we 
have also employed the same belief net- 
work and utility models we use for dis- 
play management for the problems of 
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fault detection, diagnosis, and ideal ac- 
tion selection. By using the ISP to 
distribute the results of these models 
among various applications, we are able 
to build integrated decision-making sys- 
tems based on probability and utility. 
For example, we can distribute the re- 
sults of decision-theoretic models to win- 
dow manager clients to automatically se- 
lect the optimum collection and configu- 
ration of displays for the human opera- 
tor, thereby maximizing the presentation 
of relevant information while minimizing 
irrelevant clutter. We can also distribute 
the results of the various action mod- 
els to the Flight Director, who integrates 
these actions within the current context 
into a prioritized list of actions for the 
astronauts or ground crews. 

SELMON We are working with the JPL 
selective monitoring (SELMON) project 
team to integrate SELMON techniques 
within the distributive computing mod- 
els [5, 6]. SELMON provides a collec- 
tion of sensor importance measures to 
determine which of the observered sen- 
sor values contain the most interesting 
information. There currently are seven 
sensor importance measures that can be 
applied to each value. The four empiri- 
cal measures, surprise , alarm , alarm an- 
ticipation, and value change, can be ap- 
plied directly within the ISP servers as 
part of the preprocessing, providing a 
relative information content “score” to 
each changing sensor value. This score 
is passed along with the sensor value to 
the clients. The three model-based mea- 
sures, deviation, sensitivity, and cascad- 
ing alarms, are excellent candidates for 
client-based calculations which can be 
redistributed through the servers. Since 
it is then left to the consumer clients to 
deal with this relative information score, 


we can gradually deploy this additional 
information within the clients as they 
become more “intelligent.” For exam- 
ple, we can use color changes to attract 
attention to sensor readings with high 
scores, directly influencing the user’s 
decision-making efforts while monitoring 
component behaviors. We can also ac- 
cumulate aggregate scores for entire dis- 
plays, based on the accumulation of in- 
dividual sensor scores, and prompt the 
user to concentrate his attention on that 
display because it contains more interest- 
ing information than others. These tech- 
niques fall nicely in line with the auto- 
mated display management ideas of the 
Vista project. This work also is provid- 
ing a substrate for the follow-on diag- 
nostic reasoning embedded in monitor- 
ing (DREMON) project currently being 
pursued at JSC. 

Beyond these few examples, there are also a 
variety of other distributed expert system ap- 
plications, in various stages of development, 
which assuredly will benefit from a common 
cpmmunications protocol. 


5 Information 

We encourage questions and comments about 
the ISP from both inside and outside the 
spacecraft operations community, as we hope 
to diversify into other application areas as we 
continue our refinement of this distributed- 
computing protocol. For more information, 
or to obtain the latest ISP source and pro- 
gram distribution, contact the authors at 
NAS A/ Johnson Space Center, Mail Code 
DF6, Houston, TX, 77058, or send elec- 
tronic mail to barrySrpal.rockwell.com, 
nuniklsSjscprofs.nasa.gov or nunispwS- 
j scprof s .nasa.gov. 


327 


References 


[1] Georgeff, Kinny, Hodgson, and Lee. 
PRS Operational Evaluation. SRI Inter- 
national, Project ECD 333, Menlo Park, 
CA, 1993. 

[2] Georgeff and Ingrand. Real-Time Rea- 
soning: The Monitoring and Control of 
Spacecraft Systems. Proceedings of the 
Sixth IEEE Conference on Artificial Intel- 
ligence Applications, Santa Barbara, CA, 
1990. 

[3] Horvitz, Ruokangas, Srinivas, and Barry. 
A Decision- Theoretic Approach to the 
Display of Information for Time-Critical 
Decisions: The Vista Project. Proceed- 
ings of Fourth Annual Workshop on Space 
Operations Applications and Research 
(SOAR ‘92), NASA/Johnson Space Cen- 
ter, Houston, TX, 1992. 

[4] Barry, Horvitz, Ruokangas, and Srinivas. 

Vista Goes Online: Decision-Analytic 

Systems for Real-time Decision Making 
in the Mission Control Center. Submitted 
to the Ninth Annual Goddard Conference 
on Space Applications of Artificial Intelli- 
gence, NASA/Goddard Space Flight Cen- 
ter, Greenbelt, MD, 1994. 

[5] Doyle, Chien, Fayyad, and Wyatt. 
Focused Real-Time Systems Monitoring 
Based on Multiple Anomaly Models. Sev- 
enth International Workshop on Qualita- 
tive Reasoning, Eastsound, WA, 1993. 

[6] Doyle, Charest, Rouquette, Wyatt and 
Robertson. Causal Modeling and Event- 
driven Simulation for Monitoring of Con- 
tinuous Systems. AIAA Computers in 
Aerospace 9, San Diego, CA, 1993. 


328 



